Color scannerless range imaging system using an electromechanical grating

ABSTRACT

A scannerless range imaging system includes an illumination system and an electromechanical light modulator. The illumination system illuminates objects in the scene with modulated illumination of a predetermined modulation frequency, and the modulated illumination reflected from objects in the scene incorporates a phase delay corresponding to the distance of the objects from the range imaging system. The electromechanical light modulator, which is positioned in an optical path of the reflected illumination, operates at a reference frequency that corresponds to the predetermined modulation frequency and accordingly modulates the modulated illumination reflected from the object, thereby generating a phase image from the interference between the reference frequency and the reflected modulated illumination. The system further includes an optical system that deflects the reflected illumination to the electromechanical light modulator and redirects the phase image generated by the electromechanical light modulator to an image capture section. A color image may be captured by moving the optical system out of the optical path such that reflected illumination from the object will pass directly to the image capture section without contacting the electromechanical light modulator.

FIELD OF THE INVENTION

[0001] The present invention relates to the field of three-dimensionalimage capture and in particular to techniques for modulating thereflected light in order to extract phase images and for capturing acolor texture image in conjunction with the phase image.

BACKGROUND OF THE INVENTION

[0002] Distance (or depth) information from a camera to objects in ascene can be obtained by using a scannerless range imaging system havinga modulated illumination source and a modulated image receiver. In amethod and apparatus described in U.S. Pat. No. 4,935,616 (and furtherdescribed in the Sandia Lab News, vol. 46, No. 19, Sep. 16, 1994), ascannerless range imaging system uses either an amplitude-modulatedhigh-power laser diode or an array of amplitude-modulated light emittingdiodes (LEDs) to simultaneously illuminate a target area. Conventionaloptics confine the target beam and image the target onto a receiver,which includes an integrating detector array sensor having hundreds ofelements in each dimension. The range to a target is determined bymeasuring the phase shift of the reflected light from the targetrelative to the amplitude-modulated carrier phase of the transmittedlight. To make this measurement, the gain of an image intensifier (inparticular, a micro-channel plate) within the receiver is modulated atthe same frequency as the transmitter, so the amount of light reachingthe sensor (a charge-coupled device) is a function of therange-dependent phase difference. A second image is then taken withoutreceiver or transmitter modulation and is used to eliminatenon-range-carrying intensity information. Both captured images areregistered spatially, and a digital processor is used to extract rangedata from these two frames. Consequently, the range associated with eachpixel is essentially measured simultaneously across the whole scene.

[0003] The scannerless range imaging system described above utilizes animage intensifier (specifically, a micro-channel plate) of the typeproduced by Litton Industries. The primary purpose of the intensifier isto provide a reference frequency to operate upon the modulated lightsignal from the illuminator that is reflected from the target. Bymodulating the gain of the image intensifier the reflected, modulatedlight signal is multiplied by the intensifier gain and constructive anddestructive interference is established. A primary application of thescannerless range imaging system is to enable a method of creating avirtual three-dimensional environment from photographs. While range datais an important part of this application, a so-called texture image isalso needed. The texture image should ideally be captured with identicaloptical properties as the range data to assure proper registrationbetween range and texture values. Furthermore, having a color textureimage is highly desirable for many practical and commercialapplications.

[0004] A drawback of methods using an image intensifier is that colorinformation is lost. Unfortunately for color applications, an imageintensifier operates by converting photonic energy into a stream ofelectrons, amplifying the energy of the electrons and then convertingthe electrons back into photonic energy via a phosphor plate. Oneconsequence of this process is that color information is lost. Sincecolor is a useful property of images for many applications, a means ofacquiring the color information that is registered along with the rangeinformation is extremely desirable.

[0005] One approach to acquiring color is to place a dichromatic mirrorin the optical path before the micro-channel-plate. Following the mirrora separate image capture plane (i.e., a separate image sensor) isprovided for the range portion of the camera and another image captureplane (another sensor) is provided for the color texture capture portionof the camera. This is the approach taken by 3DV Technology with theirZ-Cam product. Besides the added expense of two image capture devices,there are additional drawbacks in the need to register the two imageplanes precisely, together with alignment of the optical paths. Anotherdifficulty is collating image pairs gathered by different sources.Recognizing that the system described in the '616 patent may beimplemented in relation to a normal camera system, and, in particular,that a standard camera system may be converted into a range capturesystem by modifying its optical system, another approach is to employinterchangeable optical assemblies: one optical assembly for the phaseimage portion and a separate optical element for the color texture imageportion. This approach is described in detail in commonly assignedcopending application Ser. No. 09/451,823, entitled “Method andApparatus for a Color Scannerless Range Image System” and filed Nov. 30,1999 in the names of Lawrence A. Ray, Louis R. Gabello and Kenneth J.Repich. The drawback of this approach is the need to switch lenses andthe possible misregistration that might occur due to the physicalexchange of lens elements. There is an additional drawback in the timerequired to swap the two optical assemblies, and the effect that mayhave on the spatial coincidence of the images.

[0006] In commonly-assigned, copending U.S. patent application Ser. No.09/572,522, entitled “Method and Apparatus for a Color Scannerless RangeImaging System” and filed May 17, 2000 in the names of Lawrence A. Rayand Louis R. Gabello, a beamsplitter located in the primary optical pathseparates the reflected image light into two channels, a first channelincluding an infrared component and a second channel including a colortexture component, whereby one of the channels traverses a secondaryoptical path distinct from the primary path. A modulating element, i.e.,an intensifier, is operative in the first channel to receive theinfrared component and a modulating signal, and to generate a processedinfrared component with phase data indicative of range information. Anoptical network is provided in the secondary optical path forrecombining the secondary optical path into the primary optical pathsuch that the processed infrared component and the color texturecomponent are directed to the image responsive element. This techniqueeliminates the requirement for two image capture planes, as well as forinterchangeable optical assemblies, and allows the operator to collect afull range map with texture with a single exposure activation.

[0007] In addition to the loss of color information, and the consequentnecessity to devise techniques as described above to overcome thisdrawback, the image intensifier is a costly part and, in addition, canbe fragile. In order to reduce the cost of the scannerless range imagingsystem, a less expensive alternative technology would be attractive.Since a primary purpose of the image intensifier is to act as amodulating shutter, an alternative technology will have to perform thistask. What is needed is an alternative technology that would avoid theaforementioned limitations; in addition, it would be desirable tocapture ranging information without sacrificing color information thatwould otherwise be available for capture.

SUMMARY OF THE INVENTION

[0008] It is an object of the invention to provide an alternativetechnology to an image intensifier for the receiver modulation functionin a scannerless range imaging system.

[0009] It is a further object of the invention to capture a colortexture image as well as one or more phase images on the same imageplane for each point on the image.

[0010] The present invention is directed to achieving these objectiveswhile overcoming one or more of the problems set forth above. Brieflysummarized, according to one aspect of the present invention, ascannerless range imaging system for capturing range information of ascene includes an illumination system and an electromechanical lightmodulator. The illumination system illuminates objects in the scene withmodulated illumination of a predetermined modulation frequency, wherebythe modulated illumination reflected from objects in the sceneincorporates a phase delay corresponding to the distance of the objectsfrom the range imaging system. The electromechanical light modulator,which is positioned in an optical path of the modulated illuminationreflected from the object, operates at a reference frequency thatcorresponds to the predetermined modulation frequency and accordinglymodulates the modulated illumination reflected from the object, therebygenerating an image from the interference between the referencefrequency and the reflected modulated illumination. This captured image,which is thereafter referred to as a phase image, is used to deriverange data. An image capture section, also positioned in the opticalpath of the modulated illumination reflected from the object, capturesthe phase image.

[0011] Since the electromechanical light modulator operates viamodulating elements having reflective surfaces, the system furtherincludes an optical system having a mirror element that deflects thereflected modulated illumination upon the reflective surfaces of theelectromechanical light modulator and redirects the phase imagereflected from the reflective surfaces of the electromechanical lightmodulator to the image capture section. A color image may be captured bymoving the optical system out of the optical path such that reflectedillumination from the object will pass directly to the image capturesection without contacting the electromechanical light modulator. Apreferred electromechanical light modulator is an electromechanicalgrating.

[0012] Consequently, an advantage of the invention is that it providesboth an alternative to the intensifier and a simplified technique forcapturing a color texture image as well as one or more phase images. Afurther advantage of the invention is that it eliminates the need for anexpensive component, i.e., the intensifier, with a device that issignificantly less expensive and less fragile. The electromechanicalgrating is also lighter in weight and more compact than a micro-channelplate, and uses a lower operating voltage. Moreover, the system is ableto capture a color image in addition to the phase images without thesort of clever work-arounds shown in the prior art. The system is alsoable to operate in a continuous modulation mode or in a pulse mode.

[0013] These and other aspects, objects, features and advantages of thepresent invention will be more clearly understood and appreciated from areview of the following detailed description of the preferredembodiments and appended claims, and by reference to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014]FIG. 1 is a system level diagram of a scannerless range imagingsystem in accordance with the invention. FIG. 2 is a diagramillustrating an image bundle and related data captured by the systemshown in FIG. 1.

[0015]FIG. 3 is a diagram showing more detail of the illumination systemshown in FIG. 1.

[0016]FIG. 4 is a diagram showing more detail of the electromechanicalgrating shown in FIG. 1.

[0017]FIG. 5 is a diagram showing more detail of the image capturesystem shown in FIG. 1.

[0018]FIG. 6 is a block diagram of a known range imaging system whichcan be used to capture a bundle of images.

DETAILED DESCRIPTION OF THE INVENTION

[0019] Because range imaging devices employing laser illuminators andcapture devices including image intensifiers and electronic sensors arewell known, the present description will be directed in particular toelements forming part of, or cooperating more directly with, apparatusin accordance with the present invention. Elements not specificallyshown or described herein may be selected from those known in the art.Certain aspects of the embodiments to be described may be provided insoftware. Given the system as shown and described according to theinvention in the following materials, software not specifically shown,described or suggested herein that is useful for implementation of theinvention is conventional and within the ordinary skill in such arts.

[0020] It is helpful to first review the principles and techniquesinvolved in scannerless range imaging, as known in the prior art.Accordingly, referring first to FIG. 6, a known scannerless rangeimaging system 10 is shown as a laser radar that is used to illuminate ascene 12 and then to capture an image bundle comprising a minimum ofthree images of the scene 12. An illuminator 14 emits a beam ofelectromagnetic radiation whose frequency is controlled by a modulator16. Typically, in the prior art, the illuminator 14 is a laser devicewhich includes an optical diffuser in order to effect a wide-fieldillumination. The modulator 16 provides an amplitude varying sinusoidalmodulation. The modulated illumination source is modeled by:

L(t)=μ_(L)+η sin(2πft)  (Eq. 1)

[0021] where μ_(L) is the mean illumination, η is the modulus of theillumination source, and f is the modulation frequency applied to theilluminator 14. The modulation frequency is sufficiently high (e.g.,12.5 MHz) to attain sufficiently accurate range estimates. The outputbeam 18 is directed toward the scene 12 and a reflected beam 20 isdirected back toward a receiving section 22. As is well known, thereflected beam 20 is a delayed version of the transmitted output beam18, with the amount of phase delay being a function of the distance ofthe scene 12 from the range imaging system. Typically, in the prior art,the reflected beam 20 strikes a photocathode 24 within an imageintensifier 26, thereby producing a modulated electron streamproportional to the input amplitude variations. The gain modulation ofthe image intensifier 26 is modeled by:

[0022]M(t)=μ_(M)+γ sin(2πft)  (Eq. 2)

[0023] where μ_(M) is the mean intensification, γ is the modulus of theintensification and f is the modulation frequency applied to theintensifier 26. The purpose of the image intensifier is not only tointensify the image, but also to act as a frequency mixer and shutter.Accordingly, the image intensifier 26 is connected to the modulator 16,causing the gain of a microchannel plate 30 to modulate. The electronstream from the photocathode 24 strikes the microchannel plate 30 and ismixed with a modulating signal from the modulator 16. The modulatedelectron stream is amplified through secondary emission by themicrochannel plate 30. The intensified electron stream bombards aphosphor screen 32, which converts the energy into a visible lightimage. The intensified light image signal is captured by a capturemechanism 34, such as a charge-coupled device (CCD). The captured imagesignal is applied to a range processor 36 to determine the phase delayat each point in the scene. The phase delay term ω of an object at arange ρ meters is given by: $\begin{matrix}{\omega = \frac{4{\pi\rho}\quad f}{c}} & \left( {{Eq}.\quad 3} \right)\end{matrix}$

[0024] where c is the velocity of light in a vacuum. Consequently, thereflected light at this point is modeled by:

R(t)=μ_(L)+κ sin(2πft+ω)  (Eq 4)

[0025] where κ is the modulus of illumination reflected from the object.The pixel response P at this point is an integration of the reflectedlight and the effect of the intensification:

P=∫ ₀ ^(2π) R(t)M(t)dt=2μ_(L)μ_(M)+κπγ cos(ω)  (Eq. 5)

[0026] In the range imaging system disclosed in the aforementioned '616patent, a reference image is captured during which time themicro-channel plate is not modulated, but rather kept at a meanresponse. The range is estimated for each pixel by recovering the phaseterm as a function of the value of the pixel in the reference image andthe phase image.

[0027] A preferred, more robust approach for recovering the phase termis described in commonly-assigned U.S. Pat. No. 6,118,946, entitled“Method and Apparatus for Scannerless Range Image Capture UsingPhotographic Film”, which is incorporated herein by reference. Insteadof collecting a phase image and a reference image, this approachcollects at least three phase images (referred to as an image bundle).This approach shifts the phase of the intensifier 26 relative to thephase of the illuminator 14, and each of the phase images has a distinctphase offset. For this purpose, the range processor 36 is suitablyconnected to control the phase offset of the modulator 16, as well asthe average illumination level and such other capture functions as maybe necessary. If the image intensifier 26 (or laser illuminator 14) isphase shifted by θ_(i), the pixel response from equation (5) becomes:

P _(i)=2μ_(L)μ_(M)π+κπγ cos(ω+θ_(i))  (Eq. 6)

[0028] It is desired to extract the phase term o from the expression.However, this term is not directly accessible from a single image. Inequation (6) there are three unknown values and the form of the equationis quite simple. As a result, mathematically only three samples (fromthree images) are required to retrieve an estimate of the phase term,which is proportional to the distance of an object in the scene from theimaging system. Therefore, a set of three images captured with uniquephase shifts is sufficient to determine ω. For simplicity, the phaseshifts are given by θ_(k)=2πk/3;k=0,1,2. In the following description,an image bundle shall be understood to constitute a collection of imageswhich are of the same scene, but with each image having a distinct phaseoffset obtained from the modulation applied to the intensifier 26. Itshould also be understood that the analysis can also be performed byphase shifting the illuminator 14 instead of the intensifier 26. If animage bundle comprising more than three images is captured, then theestimates of range can be enhanced by a least squares analysis using asingular value decomposition (see, e.g., W. H. Press, B. P. Flannery, S.A. Teukolsky and W. T. Vetterling, Numerical Recipes (the Art ofScientific Computing), Cambridge University Press, Cambridge, 1986).

[0029] If images are captured with n≧3 distinct phase offsets of theintensifier (or laser or a combination of both) these images form animage bundle. Applying Equation (6) to each image in the image bundleand expanding the cosine term (i.e., P_(i)=2μ_(L)μ_(M)π+κπγ(cos(ω)cos(κ_(i))−sin(ω) sin(θ_(i)))) results in the following system of linearequations in n unknowns at each point: $\begin{matrix}{\begin{pmatrix}P_{1} \\P_{2} \\\vdots \\P_{n}\end{pmatrix} = {\begin{pmatrix}1 & {\cos \quad \theta_{1}} & {{- \sin}\quad \theta_{1}} \\1 & {\cos \quad \theta_{2}} & {{- \sin}\quad \theta_{2}} \\\vdots & \vdots & \vdots \\1 & {\cos \quad \theta_{n}} & {{- \sin}\quad \theta_{n}}\end{pmatrix}\begin{pmatrix}\Lambda_{1} \\\Lambda_{2} \\\Lambda_{3}\end{pmatrix}}} & \left( {{Eq}.\quad 7} \right)\end{matrix}$

[0030] where Λ=2μ_(L)μ_(M)π, Λ₂=κπγ cos ω, and Λ₃=κπγ sin ω. This systemof equations is solved by a singular value decomposition to yield thevector Λ=[Λ₁, Λ₂, Λ₃]^(τ). Since this calculation is carried out atevery (x,y) location in the image bundle, Λ is really a vector imagecontaining a three element vector at every point. The phase term ω iscomputed at each point using a four-quadrant arctangent calculation:

ω=tan⁻¹(Λ₃, Λ₂)  (Eq. 8)

[0031] The resulting collection of phase values at each point forms thephase image. Once phase has been determined, range ρ can be calculatedby: $\begin{matrix}{\rho \quad = {\omega \frac{c}{4\quad \pi \quad f}}} & \left( {{Eq}.\quad 9} \right)\end{matrix}$

[0032] Equations (1)-(9) thus describe a method of estimating rangeusing an image bundle with at least three images (i.e., n=3)corresponding to distinct phase offsets of the intensifier and/orilluminator.

[0033] What the present invention specifically addresses is analternative technology to the image intensifier 26, as used in the priorart. The preferred alternative technology is an electromechanicalgrating, which is in the class of electromechanical light modulators.The electromechanical grating is a device with a periodic sequence ofreflective elements that form electromechanical phase gratings. In suchdevices, the incident beam is selectively reflected or diffracted into anumber of discrete orders. Depending on the application, one or more ofthese diffracted orders may be collected and used by the optical system.

[0034] An electromechanical grating with a fast response time is abinary electromechanical grating made of suspended micro-mechanicalribbon elements as described in Bloom et al., “Method and Apparatus forModulating a Light Beam,” U.S. Pat. No. 5,311,360, issued May 10, 1994.This device, also known as a grating light valve (GLV), can befabricated with CMOS-like processes on silicon. Improvements in thedevice were later described by Bloom, et al. that included: 1) patternedraised areas beneath the ribbons to minimize contact area to obviatestiction between the ribbons and the substrate, and 2) an alternativedevice design in which the spacing between ribbons was decreased andalternate ribbons were actuated to produce good contrast (see Bloom, etal., “Deformable Grating Apparatus for Modulating a Light Beam andIncluding Means for Obviating Stiction Between Grating Elements andUnderlying Substrate,” U.S. Pat. No. 5,459,610, issued Oct. 17, 1995).An alternative electromechanical grating with a partially conformalgrating structure and a potentially higher fill factor was described byKowarz in “Spatial Light Modulator with Conformal Grating Elements,”U.S. patent application Ser. No. 09/491,354, filed Jan. 26, 2000 (CIP09/867,927 filed May 30, 2001). The disclosures of each of thesepatents, and the patent applications, are incorporated herein byreference.

[0035] As will be clear to those of ordinary skill in this art, when theelectromechanical grating is used in place of the intensifier, themethod of estimating range using an image bundle will remain essentiallythe same as described above in relation to FIG. 6 except that the outputM(t) of the modulating element represented by Eq. (2) would be modifiedto represent the effect produced by the electromechanical grating. Thismodification would carry through the remaining equations, although thebasic logic and the model would remain the same. In particular, theprinciples for range determination remain exactly the same, that is, therange is determined using an image bundle with at least three images(i.e., n=3) corresponding to distinct phase offsets of the modulatorand/or illuminator.

[0036] Referring now to FIG. 1, the overall scannerless range imaging(SRI) system is shown as a range camera comprised of a number ofsubsystems, including a controller 40, an illuminator 42, a lens/shuttercombination 44, an electromechanical grating light modulator 46, amirror 47 and an image capture subsystem 48. The image capture subsystem48 includes a photosensor, e.g., a photosensitive film or an electronicsensor, such as a charge-coupled device (CCD). The controller 40 managesthe workflow within the device, sequences the events and establishes oneor more baseline system frequencies. The illuminator 42 emitsamplitude-modulated light, preferably in the infrared band. Thecontroller 40 also has the ability to phase shift the modulation signalto the illuminator 42 relative to a reference modulation within thecontroller 40. The lens/shutter 44 controls the image focal length, thefield of view and other normal properties of photographic lenses andshutters. The electromechanical grating 46 operates at a referencefrequency that is managed by the controller 40. The purpose of theelectromechanical grating 46 is to mix a reference frequency with thereflected light from objects in the scene. The electromechanical grating46 is tuned to have maximum efficiency at the same wavelength as emittedby the illuminator. The image capture subsystem 48 records the framescaptured by the SRI (Scannerless Range Imaging) system for subsequentprocessing. In the preferred embodiment, a single image capture planeresponsive to the infrared spectrum is employed for the photosensitiveelement; in addition, if a color texture image is to be captured, thephotosensitive element must be capable of responding to light in thevisible spectrum.

[0037] The illumination and reception aspect of the SRI system shown inFIG. 1 generally operates as described in connection with the knownsystem shown in FIG. 6, that is, an output beam 43 a is directed towarda scene and a reflected beam 43 b is directed back toward the receivingsection, which includes the lens/shutter combination 44, theelectromechanical grating 46 and the image capture subsystem 48. As iswell known, the reflected beam 43 b is a delayed version of thetransmitted output beam 43 a, with the amount of phase delay being afunction of the distance of the scene from the SRI system. Unlike theprior art, the reflected beam 43 b is deflected from the system opticalpath 43 c and upon the electromechanical grating 46, thereby producing amodulated light image signal. The modulated light image signal is thendeflected back into the optical path 43 c and captured by the imagecapture subsystem 48, such as a charge-coupled device (CCD). Though notshown specifically in FIG. 1, the captured image signal is applied to arange processor of the type shown in FIG. 6 to determine the phase delayat each point in the scene. The foregoing system is sufficient tocapture a phase image. If a color texture image is also to be captured,the mirror 47 is retracted out of the optical path and the color textureimage is directly transmitted to the image capture subsystem 48.

[0038] The controller 40 is the overall device manager and has the taskof communicating with the subsystems to sequence events, to provideproper power to the devices and to provide a common synchronizingfrequency. Subsequent descriptions of each subsystem include theinterface of the subsystem with the controller. The controller is theprimary interface of a user with the system. A user need only triggerthe device once and the system creates a complete image bundle.

[0039] As shown in relation to FIG. 2, the notion of an image bundle 50is central to the range estimation method. The image bundle 50 includesa combination of images 52, 54 captured by the system as well asinformation (bundle data 56) pertinent to the individual images andinformation common to all the images. The image bundle contains twotypes of images: a set of phase images 52 related to the range captureportion of the process and a color image 54, commonly referred to as thetexture image. For the set of phase images, each phase image is acquiredwhile the illuminator 42 is operating with a phase offset from thereference frequency supplied by the controller 40. The color image 54 isacquired when the electromechanical grating 46 is inactive. Commoninformation in the image bundle data 56 would typically include thenumber of phase images in the bundle (three or more) and the modulationfrequency utilized by the camera system. Other information might be thefocal length of the lens, the size of the image plane, the number ofhorizontal and vertical pixels in the images, the field-of-view of theimaging system and/or data related to camera status at the time of theimage capture. This information will be used in subsequent processing toconvert the image values into three-dimensional positions of each pixelin space relative to the camera location and orientation.

[0040] Image specific information in the image bundle data 56 willinclude the phase offset 1 . . . N used for each (1 . . . N) of theindividual phase images 52. The image bundle 50 includes a minimum ofthree such images, each of which are monochrome. Each of the phaseimages 52 records the effect of a distinct phase offset applied toeither the illumination system 42 or the electromechanical grating 46.The additional color image 54 is an image that does not contain rangecapture components, instead containing the color texture information inthe actual image. Although this is a color image, it is preferably, butnot necessarily, the same size as the phase images 52.

[0041] The illuminator 42 shown in FIG. 3 has the primary purpose ofproducing an amplitude-modulated illumination with its phasecontrollable for generating a shift in the transmitted wave pattern foreach phase image 52 in the image bundle 50 (although, as mentionedbefore, this function may be performed by modulation of the reflectedillumination in the capture portion of the color scannerless rangeimaging system). The illuminator 42 includes a light source, which ispreferably a laser light source 60 operating in the infrared band with apower output intensity of about 0.5 watt, and a modulation circuit 62controllable through a line 64 from the controller 40 (see FIG. 1), forgenerating the requisite modulation signals of predetermined frequencywith a set of predetermined phase offsets. The emitted power should bedesigned to be maximal while maintaining compliance with Class 1 laseroperation. The laser light source 60 is preferably modulated at amodulation frequency on the order of about 10 megahertz, although thisfrequency may be adjusted to account for operating speeds of othersubsystems in the SRI system, and the preferred phase offsets, asmentioned earlier, are phase shifts θ in each phase image given byθ_(k)=2πk/3; k=0, 1, 2. The preferred wavelength of the laser light isabout 830 nm, as this wavelength provides an optimal balance betweenconcerns for eye-safety and for the typical response of the overallsystem as described. Although the laser light need not necessarily beuniformly distributed, a diffusion lens 66 is positioned in front of thelaser light source 60 in order to spread the modulated light across thedesired field of view as uniformly as possible. An alternative lightsource 60 can be a plurality of amplitude-modulated infraredlight-emitting diodes (LEDs) which are driven to generate a modulatedsignal according to the phase and frequency of a drive signal. Theilluminator 42 also includes a standard wide-band illuminator 68 that isnot modulated. This illumination source is used for normal photographicimages, e.g., for the color texture image. This illuminator device 68may be a commonly known and understood flash of a standard camerasystem, e.g., a commonly available electronic flash of the type usefulwith photographic cameras. The illuminator 42 is connected via thecontrol line 64 to the controller 40, which directs the illuminator 42to operate in either of the following modes: a) a first mode in whichthe laser is operated to illuminate the scene with a plurality (bundle)of exposures, each with a unique phase offset applied to its modulatedfrequency; and b) a second mode in which the standard wide-bandilluminator 68 is turned on and the flash is initiated by the controllerduring capture of the color texture image. If ambient light issufficient, of course, it may be unnecessary for the illuminator 42 tooperate in the second mode in order to capture a color image; in thatcase, the image capture device would be instructed to operate withoutflash. Moreover, the sequence of image capture may be reversed, that is,the second mode may be engaged before the first mode or, indeed, thesecond mode might in specific situations be engaged between the severalexposures of the first mode. The illuminator 42 also communicates withthe controller 40 via the line 64 to indicate that all systems are readyfor use. While not shown, it is preferable for the system to have somevisible indicator to the operator that the laser source is powered, asthe laser is invisible to the human eye.

[0042] Referring to the modulation system 70 shown in FIG. 4, theelectromechanical grating 46 is used to modulate the reflected light 72returning from objects illuminated by the illuminator subsystem 42. Inthe preferred embodiment, the electromechanical grating 46 is aconformal grating device of the type illustrated in the aforementionedU.S. patent application Ser. No. 09/491,354, which is incorporatedherein by reference. As conceptually shown in FIG. 4 for illustrativepurposes, a pattern of elongated reflective ribbon elements 74 aresupported at their respective ends and at several intermediate supportlocations 75 (shown in broken line to indicate their location underneaththe ribbon elements) on a substrate structure 76. The center-to-centerseparation of the intermediate supports and the mechanical properties ofthe ribbon element define the mechanical resonant frequency of theconformal grating devices in their actuated state. A signal generator 82applies a corresponding modulating voltage between the substratestructure 76 and the ribbon elements 74; as a result, an electrostaticforce generated by the voltage causes the ribbon elements 74 to deformin synchronism with the modulating frequency. In this actuated state,the incident beam 78 a is diffracted into a number of discretediffracted orders. In the unactuated state, with no applied voltagedifference, the ribbon element is suspended flat between its supportsand the incident beam 78 a is primarily reflected 78 b into thedirection of the mirror. The aperture 79 allows the reflected light topass in the return beam 78 b and blocks the diffracted orders. To obtaina greater depth of modulation at the expense of the complexity of theoptical system, the reflected light can be blocked and the diffractedorders can be passed through the system. In actual practice, the area ofthe electromechanical grating 46 would be designed to fill the imagespace with thousands of ribbon elements. Furthermore, theelectromechanical grating 46 responds to light at preferred wavelengths,and is tuned to the same wavelength as the infrared light produced bythe laser source 60. Further details of the electromechanical grating,including more detailed renderings of the actual structure of thedevice, can be found in the aforementioned U.S. patent application Ser.No. 09/491,354, which is incorporated herein by reference.

[0043] The system 70 is controlled through a control line 80 attached tothe system controller 40 and through the signal generator 82 driving theelectromechanical grating 46. (While shown as a separate component, thesignal generator 82 may be integrated together with theelectromechanical grating in a common CMOS-like element.) The signalgenerated by the signal generator 82 is in synchronization with theoverall system frequency provided by the controller 40 and serves as areference waveform that beats against the reflected waveform. Typically,the electromechanical grating 46 has a natural resonance frequencybetween 5 and 15 MHz. For an electromechanical grating 46 that issufficiently damped, the signal generator 82 periodically drives thedevice between the unactuated state and the actuated state at afrequency lower than resonance, generating light intensity modulation atthis frequency. Alternatively, for an underdamped electromechanicalgrating 46, the signal generator 82 drives the device at resonance. Inthis resonant mode of operation, the ribbon elements 74 oscillatesymmetrically about their unactuated position and generate sinusoidallight intensity modulation at twice the resonance frequency.

[0044] Since the electromechanical grating 46 is activated when a phaseimage is to be captured and is inactive when a color texture image iscaptured, the retractable mirror 47 is operated to position the mirrorin-line such that one of its facets deflects the light 78 a toward thesurface of the electromechanical grating 46 when phase images are beingcaptured. The returned light 78 b is then modulated by theelectromechanical grating 46 and reflected back to the other facet ofthe retractable mirror 47, which redirects the light toward the imagecapture subsection 48. When the color texture image is being collected,the mirror is retracted sufficiently far in order to eliminate it frominterfering with the incoming light. As a result, the system is able torecord visible light necessary for the color texture image. While manydifferent types of devices may be used to toggle the mirror back andforth, FIG. 4 shows a solenoid 84 connected to the mirror 47 to drivethe mirror in the direction of the arrows 86 when so instructed by thecontroller 40. The electromechanical grating may operate either in thecontinuous modulation mode as described herein or in a pulse mode. Ifthe pulse mode operation is desired, then the image capture and rangeestimation approach follows a waveform analysis based on gating themodulation at preselected points. This has been disclosed in priordisclosures, such as U.S. Pat. No. 5,081,530 (which is incorporatedherein by reference), for range cameras and details of the approach arenot included here.

[0045] In referring to FIG. 5, the image capture subsystem 48 of the SRIdevice shares the properties common to most camera systems: a lens 90and shutter 92 integral with the image capture subsystem (or,alternatively, the separate lens/shutter 44 shown in FIG. 1), an imagecapture plane 94 and an image storage capability 96. As mentioned above,the image capture plane 94 includes a photosens or, e.g., aphotosensitive film or an electronic sensor, such as a charge-coupleddevice (CCD). If a film is located in the image capture plane 94, thenthe film itself is the image storage capability 96. On the other hand,if an electronic sensor is located in the image capture plane 94, thenthe image storage capability 96 is an electronic storage device, such asresident solid state memory (e.g., RAM or ROM) or a removable memory(e.g., a memory card). In addition, this subsystem 48 may alsoresponsible for collecting and storing some or all of the data portionof the image bundle described above. The controller 40 of the devicetriggers the image capture subsystem 48 on a control line 98. In turnthe system opens the shutter 92, records the image on the image captureplane 94 and stores an image in the image storage 96 prior to asubsequent exposure period. Furthermore, the system could provide arange processor of the type shown in FIG. 6 to extract the images fromthe image bundle for external processing, though this is not explicitlyindicated in FIG. 5. This range processor could be integral with the SRIcamera or located in an ancillary processor.

[0046] The invention has been described with reference to a preferredembodiment. However, it will be appreciated that variations andmodifications can be effected by a person of ordinary skill in the artwithout departing from the scope of the invention.

[0047] Parts List 10 range imaging system 12 scene 14 illuminator 16modulator 18 output beam 20 reflected beam 22 receiving section 24photocathode 26 image intensifier 30 microchannel plate 32 phosphorscreen 34 capture mechanism 36 range processor 40 controller 42illuminator 43 a output beam 43 b reflected beam 43 c optical path 46electromechanical grating 47 mirror 48 image capture subsystem 50 imagebundle 52 phase images 54 texture (color) image 56 bundle data 60 laserlight source 62 modulation circuit 64 line to the controller 66diffusion lens 68 wide-band illumination 70 modulation system 72reflected light 74 ribbon elements 75 support locations 76 substratestructure 78 a incident beam 78 b return beam 79 aperture 80 line fromthe controller 82 signal generator 84 solenoid 86 direction arrow 90lens 92 shutter 94 image capture plane 96 image storage capability 98control line

What is claimed is:
 1. A scannerless range imaging system for capturingrange information of a scene, said system comprising; an illuminationsystem for controllably illuminating the scene with modulatedillumination of a predetermined modulation frequency, whereby modulatedillumination reflected from an object in the scene incorporates a phasedelay corresponding to a distance of the object from the range imagingsystem; an electromechanical light modulator positioned in an opticalpath of the modulated illumination reflected from the object, whereinthe electromechanical light modulator operates at a reference frequencythat corresponds to the predetermined modulation frequency andaccordingly modulates the modulated illumination reflected from theobject, thereby generating a phase image from the interference betweenthe reference frequency and the modulated illumination reflected fromthe object; and an image capture section positioned in the optical pathof the modulated illumination reflected from the object for capturingthe phase image, whereby the range information is derived from the phaseimage.
 2. The range imaging system as claimed in claim 1 wherein theelectromechanical light modulator includes modulating elements havingreflective surfaces, said system further including an optical systeminterposed in the optical path and comprised of a mirror element thatdeflects the reflected illumination upon the reflective surfaces of theelectromechanical light modulator and redirects the phase imagegenerated by the reflective surfaces to the image capture section. 3.The range imaging system as claimed in claim 2 wherein the opticalsystem is movable out of the optical path such that reflectedillumination from the object will pass directly to the image capturesection without contacting the electromechanical light modulator.
 4. Therange imaging system as claimed in claim 3 wherein the image capturesection captures at least one phase image used to derive a range imageand another image of reflected unmodulated illumination corresponding tocolor in the scene when the optical system is moved out of the opticalpath.
 5. The range imaging system as claimed in claim 1 wherein theelectromechanical light modulator is an electromechanical grating. 6.The range imaging system as claimed in claim 1 wherein the image capturesection includes a photosensitive film for capturing the phase image. 7.The range imaging system as claimed in claim 1 wherein the image capturesection includes an electronic image sensor for capturing the phaseimage.
 8. The range imaging system as claimed in claim 1 wherein theimage capture section captures a plurality of phase images correspondingto the reflected modulated illumination, wherein each phase imageincorporates the effect of the predetermined modulation frequencytogether with a phase offset unique for each image.
 9. The range imagingsystem as claimed in claim 8 wherein each unique phase offset θ is givenby θ_(i)=2πi/3; i=0, 1,
 2. 10. The range imaging system as claimed inclaim 8 wherein the image capture section further comprises means forstoring the phase images as a bundle of associated images.
 11. The rangeimaging system as claimed in claim 1 wherein the illumination systemincludes either a laser illuminator or an array of light emitting diodesfor producing the modulated illumination.
 12. The range imaging systemas claimed in claim 1 wherein the predetermined modulation frequency isin the infra-red spectrum.
 13. A color scannerless range imaging systemfor capturing both color and range information of a scene, said systemcomprising; an illumination system for controllably illuminating objectsin the scene with modulated illumination of a predetermined modulationfrequency, whereby modulated illumination reflected from an object inthe scene incorporates a phase delay corresponding to a distance of theobject from the range imaging system; a modulation section including (a)an electromechanical light modulator positioned in an optical path ofthe modulated illumination reflected from the object, wherein theelectromechanical light modulator operates at a reference frequency thatcorresponds to the predetermined modulation frequency and accordinglymodulates the modulated illumination reflected from the object, therebygenerating a phase image from the interference between the referencefrequency and the modulated illumination reflected from the object,wherein said phase images are used to derive the range information, and(b) an optical system located in the optical path to transmit thereflected modulated illumination to the image capture section via theelectromechanical grating when a phase image is to be captured; an imagecapture section positioned in the optical path of the modulatedillumination reflected from the object for capturing a plurality ofimages thereof, including (a) at least one phase image corresponding tothe reflected modulated illumination and (b) at least one other image ofreflected unmodulated illumination corresponding to color in the scene;and a controller for activating the modulation section when a phaseimage is to be captured and inactivating the modulation section when acolor image is to be captured.
 14. The range imaging system as claimedin claim 13 wherein the electromechanical light modulator includesmodulating elements having reflective surfaces, said optical systemfurther includes a mirror element having facets that deflect thereflected modulated illumination upon the reflective surfaces of theelectromechanical light modulator and redirect the phase image generatedby the reflective surfaces of the electromechanical light modulator tothe image capture section.
 15. The range imaging system as claimed inclaim 14 wherein the controller inactivates the modulation section bymoving the optical system out of the optical path such that thereflected unmodulated illumination corresponding to color in the scenewill pass directly to the image capture section without contacting theelectromechanical light modulator.
 16. The range imaging system asclaimed in claim 13 wherein the electromechanical light modulator is anelectromechanical grating.
 17. The range imaging system as claimed inclaim 13 wherein the image capture section includes a photosensitivefilm for capturing the phase image.
 18. The range imaging system asclaimed in claim 13 wherein the image capture section includes anelectronic image sensor for capturing the phase image and the colorimage.
 19. The range imaging system as claimed in claim 13 wherein theimage capture section captures a plurality of phase images correspondingto the reflected modulated illumination, wherein each phase imageincorporates the effect of the predetermined modulation frequencytogether with a phase offset unique for each image.
 20. The rangeimaging system as claimed in claim 19 wherein each unique phase offset θis given by θ_(i)=2πi/3; i=0, 1,
 2. 21. The range imaging system asclaimed in claim 19 wherein the image capture section further comprisesmeans for storing the phase images and the color image as a bundle ofassociated images.
 22. The range imaging system as claimed in claim 13wherein the illumination system includes either a laser illuminator oran array of light emitting diodes for producing the modulatedillumination.
 23. The range imaging system as claimed in claim 13wherein the predetermined modulation frequency is in the infra-redspectrum.
 24. The range imaging system as claimed in claim 13 whereinthe illumination system also emits unmodulated illumination and thereflected illumination includes unmodulated illumination originatingwith the illumination system and reflected from objects in the scene.25. The range imaging system as claimed in claim 13 wherein thereflected illumination includes unmodulated illumination from ambientillumination reflected from objects in the scene.